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Abstract. In the case of finite groups, a separating algebra is 
f*^; , a subalgebra of the ring of invariants which separates the orbits. 

■ Although separating algebras are often better behaved than the 

ring of invariants, we show that many of the criteria which imply 

j^JT) 1 the ring of invariants is non Cohen-Macaulay actually imply that 

^ , no graded separating algebra is Cohen-Macaulay. For example, we 

■ show that, over a field of positive characteristic p, given sufficiently 

many copies of a faithful modular representation, no graded sep- 
arating algebra is Cohen-Macaulay. Furthermore, we show that, 
for a p-group, the existence of a Cohen-Macaulay graded separat- 
ing algebra implies the group is generated by bireflections. Ad- 
ditionally, we give an example which shows that Cohen-Macaulay 

■ separating algebras can occur when the ring of invariants is not 
r- 1 \ Cohen-Macaulay. 

+-> 
C3 



1. Introduction 

Q\ ' Let G be a finite group, and let V be a finite dimensional repre- 

sentation of G over a field k of characteristic p > 0. We say V is a 
modular representation if p divides \G\. We write k[V] for the sym- 
metric algebra S(V*) on the vector space dual of V. It is a polynomial 
ring with the standard grading. The action of G on V induces an ac- 
tion on k[V]: for / G V* and v G V, the action of a G G is given by 
(a • f)(v) = /(cr^ 1 ■ v). The ring of invariants, denoted k[V] G , is the 
ring formed by the elements of k[V] fixed by G?. Since the group action 
preserves degree, kfV^] is a graded subalgebra of k[V]. 
■ Let k be an algebraic closure of the field k, and let V = k (git V. As 

k[V] C kigikkfy] = k[V], any / in k[V) can be considered as a function 
V — > k. The action of G on V extends to an action of G on V, and so 
k[V] G C k[Vf. 

By definition, elements of k[V] G are constant on G?-orbits. Accord- 
ingly, if an invariant / takes distinct values on elements u, v G V, then 
these elements belong to distinct orbits, and we say / separates u and v. 
A geometric separating set is then a set of invariants which separates 
exactly the same points of V as the whole ring of invariants. As G 
is finite, the ring of invariants separates orbits in V [TTJ Lemma 2.1]. 
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Hence, a geometric separating set is a set of invariants which separates 
the orbits of the G-action on V. 

The study of separating invariants, which has become quite popular 
in recent years, was initiated by Derksen and Kemper (0 Section 2.3] 
and [24J). They defined a separating set as a set separating the same 
orbits in V as the whole ring of invariants [8, Definition 2.3.8]. For k 
algebraically closed, geometric separating sets coincide with separating 
sets, but in general, a separating set is not always a geometric separat- 
ing set (see Example 12. 5p . On the other hand, A C k[V] G is a geometric 
separating algebra if and only if Ik ®t A C k[V] G is a separating alge- 
bra. Moreover, if A is graded, so is k (gi^ A, and the two rings have the 
same dimension and depth. Furthermore, the extension A C k[V] G is 
integral if and only if k£g>ky4 C k[V] G is integral. Thus, it often suffices 
to write proofs for separating sets over algebraically closed fields. Note 
that this works only because we are interested in geometric separating 
algebras. 

Many defects of invariant rings disappear when one considers sepa- 
rating invariants. The ring of invariants is not always finitely gener- 
ated (for non-reductive groups) [8, Example 2.1.4, due to Nagata], but 
there always exist finite geometric separating sets 0, Theorem 2.3.15]. 
Over algebraically closed fields, there is an upper bound on the size of 
minimal separating sets, depending only on the dimension of the rep- 
resentation [121 Proposition 5.1.1]. Polarization, a classical method for 
obtaining vector invariants in characteristic zero, extends, for separat- 
ing invariants, to all characteristics [TTl[TU]. For G finite, the Noether 
bound holds for separating invariants in all characteristics: although 
they may not generate the ring of invariants, the invariants of degree 
at most \G\ always form a geometric separating set [HI Section 3.9]. 

Graded separating algebras are very closely related to the ring of 
invariants: 

Proposition 1.1. If A C k[V] G is a graded geometric separating al- 
gebra, then A C k[V] G is an integral extension, and A is a finitely 
generated k-algebra. 

Proof. We may assume k is algebraically closed. By [2H Lemma 1.3], 
the extension A C k[V] G is integral. The finite generation of A as a 
k-algebra then follows from that of k[V] by Newstead [26l P- 52, (II)], 
or in the manner of [H Proof of Theorem 1.3.1]. □ 

Proposition 1.2. Suppose p > 0. If A C k[y] G is a graded subalgebra, 
then A is a geometric separating algebra if and only if k[V] G is the 
purely inseparable closure of A in k[V], that is, 

k[V] G = {f E k[V] | for some m, f pm 6 A}. 

Proof. For k algebraically closed, see [9], Remark 1.3]. The proof is 
an application of a result of van der Kallen J28J Sublemma A. 5.1] (see 
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the extended proof in [27]). A variation of the same argument first 
appeared in [TTJl Theorem 6]. For / G k|V] G and k arbitrary, we have 
that f pm 6 for some m. The stability of the rank of matrices 

under field extensions implies f pm G A. □ 

Remark 1.3. Propositions [1J] and [T72] hold for rational representations 
of reductive groups. 

Kemper [21] exploited this close relationship to compute the invari- 
ants of reductive groups in positive characteristic. On the other hand, 
Dufresne [13] showed that the existence of polynomial or complete in- 
tersection separating algebras imposes strong conditions on the repre- 
sentation. The present paper is in the latter vein. We show that, in 
many instances, conditions which ensure that the ring of invariants is 
non Cohen-Macaulay, in fact imply that no graded geometric separat- 
ing algebra is Cohen-Macaulay. We thus provide a (partial) negative 
answer to Kemper who asked if Cohen-Macaulay separating algebras 
should always exist |25]. Notably, we show: 

Theorem 1.4. IfV is faithful and modular, then there exists r > 1 such 
that, for all k, every graded geometric separating algebra in k[V® k } G has 
Cohen-Macaulay defect at least k — r — 1. In particular, for k > r + 1, 
no graded geometric separating algebra in k[V® h ] G is Cohen-Macaulay. 

An element a of G acts as a bireflection on V if its fixed space is of 
codimension at most 2 in V. 

Theorem 1.5. Let G be a p-group. If there exists a graded geomet- 
ric separating algebra in k[V] G which is Cohen-Macaulay, then G is 
generated by elements acting as bireflections. 

Theorem 11.51 fits well with [131 Theorem 1.3]. In the important spe- 
cial case of ]9-groups, we obtain that G is generated by bireflections 
from a much weaker hypothesis: the existence of a Cohen-Macaulay 
rather than a complete intersection graded geometric separating alge- 
bra. This mirrors the situation for invariant rings ([221 Corollary 3.7] 
and (2T1 Theorem A] ) . Example 12.101 shows that not only the converse 
of Theorem 11.51 but also the converses of [T3l Theorem 1 . 1 and Theo- 
rem 1.3], are not true. 

In Section [2j we extend the methods introduced in [22] to prove our 
main results. Section [3] concentrates on the alternating group A±. We 
conclude in Section H] with a discussion of the general situation and 
examples which show that the depth of graded geometric separating 
algebras can be both larger and smaller than that of the corresponding 
invariant ring. 

Acknowledgements. This paper was prepared during visits of the 
first and second authors to TU Munchen, and of the second and third 



4 



EMILIE DUFRESNE, JONATHAN ELMER, AND MARTIN KOHLS 



authors to Universitat Heidelberg. We thank Gregor Kemper, B. Hein- 
rich Matzat, and MATCH for making these visits possible. Finally, we 
thank the anonymous referees for helpful suggestions, in particular for 
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2. The Cohen-Macaulay Defect of Separating Algebras 

Let /I C be a finitely generated graded subalgebra, and let 

A + denote its maximal homogeneous ideal. Homogeneous elements 
ai,...,a,k in A + form a partial homogeneous system of parameters 
(phsop) if they generate an ideal of height k in A. If additionally 
k = dim A, then they form a homogeneous system of parameters (hsop). 
Noether's normalization theorem guarantees that a hsop always ex- 
ists. If, for i = 1, . . . , k, the element a, is not a zero divisor in 
Aj (di, . . . , cii-i)A, then the elements aj., . . . , a* form a regular sequence. 
Every regular sequence is a phsop. We say A is Cohen-Macaulay when 
every phsop is a regular sequence. The depth of a homogeneous ideal 
/ C A + , written depth^(J), is the maximal length of a regular sequence 
in /. Note that the height of /, ht(J), is equal to the maximal length 
of a phsop in I. We write depth (A) := depth A (v4 + ), and define the 
Cohen-Macaulay defect of A to be cmdef(v4) := dim A — depth(A). 
Thus, A is Cohen-Macaulay precisely when cmdef A = 0. 

In Theorem l2.lt we relate the Cohen-Macaulay defect of graded geo- 
metric separating algebras to the n-th cohomology group H n (G, k[V]). 
For the theory of these groups for arbitrary n, we refer to [21 EH]- Since 
we have n = 1 in most applications of this theorem, we construct this 
group now. 

Let W be a representation of the group G over the field k. A 1-cocycle 
is a map g \ G — > W, o~ i — > g a such that g aT = ag T + g a , for all a, r G G. 
We write Z 1 (G,W) for the additive group of all 1-cocycles. For each 
w G W, the map given by o \— > (er — l)w := aw — w is a 1-cocycle, 
which is called a 1-coboundary. The coboundaries form the subgroup 
B 1 (G,W) of Z 1 (G,W). The first cohomology group of G with coeffi- 
cients in W is the quotient group H\G,W) := Z 1 (G,W)/B 1 (G,W). 
A cocycle g is nontrivial if and only if its cohomology class g+B 1 (G, W) 
is nonzero in H l (G,W). Similar definitions hold for H n (G, W). We 
will sometimes abuse notation by using the same symbols for cocycles 
and cohomology classes. 

The group H n (G, k[V]) has a natural graded k[V] -module struc- 
ture. For a homogeneous g G H n (G, k[V]), its annihilator in k[V] G , 

Arm m a(g) := {a G k[V] G : ag = G H n (G,k[V})}, 

is a homogeneous ideal. If p > 0, the m-fold Frobenius homomorphism 
k[V] -> k[V], f i-> f pm , induces a map # n (G, k[V}) -> iP(C7, k[F]). 
This map is a homomorphism of abelian groups, but not of k-vector 
spaces. We write g pm for the image of an element g G H n (G,k[V]) 



THE C-M PROPERTY OF SEPARATING INVARIANTS 



5 



under this map. In particular, for g G H l (G, k[V]), the cohomology 
class g pm G k[V]) is given by the cocycle a t— > (5 , fT ) p . 

Over fields of characteristic zero, and in the non-modular case in 
general, the ring of invariants is always Cohen-Macaulay |20j. In par- 
ticular, there will always be a Cohen-Macaulay geometric separating 
algebra. Accordingly, from now on we assume V is modular. 

Our most general statement generalizes [221 Corollary 1.6] (see also 
Proposition 6]) to the case of separating algebras: 



Theorem 2.1. Let n > 1 be the smallest integer such that there exists 
a homogeneous g G H n (G,k[V]) such that g pm is nonzero for every 
m > 0. If A is a graded geometric separating algebra in k[V] G , then 
I := Ann^(g), the annihilation ideal of g in A, has depth at mostn + 1. 
Furthermore, if Aim^\a(g) has height k, then A has Cohen-Macaulay 
defect at least k — n — 1 . 

Since the case n = 1 suffices for most of our applications, we give 
an additional more elementary proof of the first part. Without loss of 
generality, we assume k = k in both arguments. 



Proof of the case n — 1 . By Proposition 11.21 there exists a p-power q 
such that (kjV] ) 9 C A. Suppose, for a contradiction, that depth A (J) 
is at least 3. Hence, there exists an A-regular sequence ai,a 2 ,a 3 in /. 
Since a^g = 0, there are bx, b 2 , b 3 G k[V] such that 

a i9a — (c — for all a G G, i = 1, 2, 3. 

Set for 1 < i < j < 3. For all i, j, Uy is invariant, 

and so belongs to A. Since a\ 1 a\ 1 a\ forms an A-regular sequence 
[lU Corollary 17.8 (a)], and since 



q q q q . q q 

a l U 23 ~ a 2 U 13 ' a 3 U 12 



ai a 2 a 3 
cti a 2 a 3 
h b 2 b 3 



0. 



it follows that u\ 2 G [a\ , a\)A. Thus, there exist f\, f 2 G A such that 

u\ 2 = a\b\ - a\b\ = f x a\ + f 2 a\. 

As ax, a 2 is a phsop in A, it is also a phsop in its integral extension k[V], 
and thus ax, a 2 are coprime in k[V]. From a\{b\ — f\) = a 2 (f 2 + b\), it 
follows that a\ divides f 2 + bf in k[V]. Therefore, there exists h G k[V] 
such that a\h = f 2 + b\ . Hence, for every o~ G G, we have 

a\{a - l)h = ((a - l)bx) q = (a l9(7 y = a\g% 

that is, g q a = (a — l)h, a contradiction since g q is nonzero. Thus, 
depths (/) < 2. □ 

Proof of the general case. For some m > 0, we have (k[V] G ) pm C A. 
For each i, H l (G, k[V]) is finitely generated as a k[V] G -module. There- 
fore, for each < % < n, there exists some m(i) such that a pm 1 = 
for all a G H l (G,k[V}). Set q := p m , where ml is the maximum 
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of m,m(l),m(2), . . . ,m(n — 1). Assume, for a contradiction, that 
depth j4 (J) > n + 2, and let a±, a 2 , . . . , a n+2 be an A- regular sequence 
in /. Consider the bar resolution of Z as a ZG-module: 

d n +i y d n v 9 n -l di v 8q ™ „ 
... — > A n — > A n _! — > . . . — > A — > IL — > U, 

where X = ZG, and <9o is the augmentation map. The cohomol- 
ogy class g is represented by a G HoniK;(X n , k[V]) such that 
M{i}9 n+ i = 0, that is, U{\] G Z n (G, k[V]) (the notation will become 
clear later). For each i, a^g = 0, so there is h^y G Hom^G(X n _i, k[V]) 
such that ajM{!} = h^d n . We next define, for each 2 < r < n+ 1, and 
for each ordered r-subset J := {j(l), j(2), . . . , j(r)} of {1, . . . ,n+ 2}, 
a homomorphism uj G Hom ZG (X n _ r+1 ,lk[y]) such that Ujd n ^ r+2 = 0, 
that is, uj G Z n_r+1 (C7, k[V]). For 2 < r < n, the definition of g im- 
plies u q j G 5"- r+1 (G, k[1/]), so there is a map hj G Hom ZG (X ri _ r , 
satisfying Uj = hjd n ^ r+ i. We now define u j, and thus hj, by induction 
on r: 

r 

(1) «J := E(- 1 ) ma S 2/l AO«}- 

i=i 

Next, we show that ujd n - r+ 2 = for 2 < r < n + 1. For r = 2, we get 

U{j(i),j(2)}d n = a j{1) h {jm d n - a j(2 )h {j{1)} d n 

= a j{1) a j{2) u {1} - a j(2 )a j{1) u {1} = 0. 

For 2 < r < n + 1, we obtain similarly: 

r 

(2) ujd n „ r+2 = E(- 1 ) 1+1 <« 2m AO«} = °> 

i=l 

since the middle term is 
j=l \fc=l 



q r ~ 2 ) 

k=i+l 

2Jk l ) a j(i) a j(k) n J\{j(i),j(k)} + 2-J* y > a j(i) a j{k) n J\{j(i),j(k)}> 

l<k<i<r l<i<k<r 



which equals zero. When r = n + 1, Uj G Z°(C7, k[V]). It follows that 
uj(i) G kfV^] 61 , which implies uj(i) g e A (i <E G is the neutral element). 
Therefore, for each 1 < i < n + 2, we have itl 2 n+2 \\uy( L ) e ^- The 
second equality in (T5]) is also valid for J = {1, 2, . . . , n + 2} (r = n + 2), 
that is, 

n+2 

E(- 1 )* 4 " la f«{i A ..^}\«w = o- 



THE C-M PROPERTY OF SEPARATING INVARIANTS 7 

As of, ... , a^ +2 is A-regular, u q {1 ^ +1} (i) G (af , . . . , a 9 „ +1 )A. Thus 
there exist fi, f 2 , ■ ■ ■ , fn+i e Hom ZG (X ,A) (in particular, = 0) 
such that n+1 j = £)i=i a i /*• Substituting (Q]) for W{i,... jn +i} yields 

n+l 

E(aT)((-l) 4+1 /i{ lf ...,^ 1}VW -/,) = 0. 

8=1 

Since , . . . , a n+1 is a phsop in A, it is also a phsop in k[V], and so 
((- 1 ) n/i ?i,...,n } - /n+i) W e (of, «f> ■ • • > OkM. ^ follows that 



i=l 

for some / 1; i 2 , . . . , i n G Hom ZG (X , k[V]) = k[V]. We next apply <9i to 
this expression. For n = 1, we have a\u q ^ = a\l\dx, or alternatively 
= ^ 1 ^ a contradiction to 7^ 0. Now assume that 

n > 2. Applying d\ leads to 



( 3 ) M {i,...,n } = ^2afl i d 1 . 

i=i 

For 2 < r < n, we prove by reverse induction that there exist elements 
li, I2, ■ ■ ■ , l r E HomzG(A n _ r , k[V]) such that 



(4) •cr J =E«rw 



n— r+1 • 

i=l 

The case r = n is covered by ©. Suppose it?-^ r+1 , = Yji=i a \ Vfin- 
for some l[, 1' 2 , . . . l' r+1 G Homzop^n-r-i, k[V]). Using (CD), we obtain 

r+l r+l 

Vr-iv + W l+1 ~ r -V/f9 

/ A L ) a i n h r+l>\ffl / j U i L i u n-r-, 

i=l i=l 

and rearranging yields 

V a qn ((-l) l+1 h qn+1 ~ r -I'd )-0 

/ j a i \\ L > ll {l,...,r+l}\{i} H U n-r)—V- 



Since a\ , . . . , a q r+l is a phsop for k[V], we have 

^{l,...,r} _ { — ^-Y K+l^n-r = a i h 

i=l 

for some l±,l2, ■ ■ ■ ,l r G HomzG(A n _ r , k[V]). Here we have used that 
X n ^ r is a free ZG-module. Applying G>„_ r +i to this expression gives us 

r 

h qn+1 ~ r R -X^n qn lf) 
{l,...,r} n— r+l — / j i i i u n—r+li 

i=l 
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which implies fll]), as required. 

When r = 2, Equation (|4J) reads Ur 12 \ = a? hd n -i + «2 hd n -i, 
where ii, l 2 G Hom ZG (X n _ 2 , k[V]). Substituting (pQ) for W{i,2}, we obtain 
a ? (^{2} — ^i^n-i) = a t (^{i} +^2^ n -i)- For the same reasons as before, 
we have h^+hdn-i = a\ I for some I G HomzG(X n _i, k[V]). Applying 
d n gives us a\ u 9 ^ = a\ ld n . Hence, wl, = Z9 n G B n (G,k[V]), a 
contradiction to g 9 " ^ 0. Therefore, depth j4 (J) < n + 1. 

Finally, if ci,...,Cfc G Ann k [y]G(g) forms a phsop in k[V] G , then 
c\, . . . , c\ G / forms a phsop in A, and so J has height at least k. The 
graded analogue of [5, Exercise 1.2.23] implies the Cohen-Macaulay 
defect of A is at least ht a(I) — depth A (J) > k — n — 1. □ 

Lemma 2.2. If g G H n (G, k) zs nonzero, then g pm is nonzero for all 
m > 0. 

Proof. For n = 1, this is clear since elements of H l (G, k) are group 
homomorphisms G — > (k, +). For arbitrary n, by the Universal Coeffi- 
cient Theorem [TBI page 30], H n (G, k) = fP(G, F p ) ® Fp k. We have # = 
9i®"^i f° r some ^ G H n (G, F p ) and Aj G k. Without loss of gener- 
ality, we can assume that the set {Aj : i G /} is F p -linearly independent 
and gi is nonzero for all i. Then g pm = 9i ®K = 9i®^i > 
since the m-fold Frobenius homomorphism induces the identity map on 
F p , and thus also on F p ). Therefore, as {X p : i G /} is still F p - 

linearly independent, g pm is still nonzero. □ 

Remark 2.3. For n > 1, Theorem 12. H is new even in the case A = k[V] G . 

Example 2.4. Let G C (k, +) be a finite nontrivial subgroup. Consider 
the threefold sum of the 2-dimensional representation V of G over k 
given by a ^ Write k[V® 3 ] = k[ari, j/i, x 2 , y 2 , z 3 , 2/3] and 

p for the induced G-action. The map g : G — > k, a 1— > cr yields a 
nonzero element in -ff 1 (G, k). For all u 6 G, we have Xjg CT = (p(er) — 
1)3/1, that is, a^p is trivial. Since xi,^,^ form a phsop in Hc[V® 3 ], 
Theorem 12.11 implies that no graded geometric separating algebra in 
k[y®3jG ig c hen-Macaulay. < 

The following example shows that Theorem 12 . 1 1 applies only to graded 
geometric separating algebras: 

Example 2.5. Let V be the permutation representation of the cyclic 
group C4 = (a) of order 4 over the field F 2 . Consider the Grinvariants 
Ci := X1+X2+X3+X4, c 2 := X1X3+X2X4, c 3 := X1X2+X2X3+X3X4+X1X4, 
and C4 := X1X2X3X4. The action of G4 on V partitions its 16 elements 
into 6 orbits, which one can check are separated by ci, c 2 , C3, and C4. 
As ci, C2, C3, C4 form a hsop in F 2 [V r ], the subalgebra F 2 [ci, 02,03, C4] is a 
polynomial graded (non geometric) separating algebra. In particular, 
it is Cohen-Macaulay. 
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On the other hand, if g : C4 — > F 2 is the nontrivial cocycle given by 
o~ l 1 — > i mod 2, then C\g a % = {a 1 — l){x\ + x 3 ), c 2 g a i = (cr l — l)(xiX 3 ), 
and c 3 g a i = (V - l){x x x A + x 2 x 3 ), that is, Ci,c 2 ,c 3 e Aan W2 ^c 4 {g) . 
Since ci,C2,c 3 ,c 4 form a hsop in F 2 [V], by Theorem 12.11 no graded 
geometric separating algebra in F 2 [V] C ' 4 is Cohen-Macaulay. < 

Using Theorem 12.11 we can quickly generalize several results which 
were consequences of its analogue [22, Corollary 1.6]. A first conse- 
quence is the following generalization of [HI Corollary 21]: 

Corollary 2.6. Assume G contains a normal subgroup N of index p 
(for example, G is a p-group). Then for any faithful representation 
V , every graded geometric separating algebra in k\V® k ] G has Cohen- 
Macaulay defect at least k — 2. 

Proof. Since G/N = (F p , +), there is a nonzero element in if 1 (G,k). 
As V is faithful, the fixed subspaces of nonidentity elements in G have 
codimension at least k in V® k . The result follows from Lemma [2.71 □ 

Lemma 2.7. Suppose V is faithful. If the fixed subspace of every ele- 
ment of order p in G has codimension at least k in V , then for any ho- 
mogeneous g G H n (G, k[V]), the ideal Ann^yp^) has height at least k. 

Therefore, if there exists a homogeneous g G H n (G,k[V}) satisfying 
the hypotheses of Theorem \2.1\ then every graded geometric separating 
algebra in k[K] G has Cohen-Macaulay defect at least k — n — 1. 

Proof. We may assume k = k. By Kemper [2"2l Lemma 2.1] there exist 
(in general, non- homogeneous) elements a%, . . . , e klV]^ such that 
aiH n (G,k[V]) = 0, for all i, and ht(ai, . . . , a*.) = k. It follows that 
Ann k [y]G(g) has height at least k. □ 

Proof of Theorem \1.4\ By [31 Theorem 4.1.3], there is a number r such 
that H r (G, k) 7^ 0. Thus by Lemma [2.21 for any k, there is a minimal 
number n < r such that the hypotheses of Theorem 12.11 are satisfied 
for V® k . The same argument as in Corollary 12.61 shows that A has 
Cohen-Macaulay defect at least k — n — 1 > k — r — 1 . □ 

Next, we generalize three results of [22] . Note that since, for us, 
elements acting trivially are bireflections, we do not need to assume 
that V is faithful. 

Corollary 2.8. If G has a normal subgroup N of index p which con- 
tains all elements acting as bireflections on V , then no graded geometric 
separating algebra in k[V} G is Cohen-Macaulay. 

Proof. The proof of [22} Theorem 3.6] shows that the hypotheses of 
Theorem 12.11 are fulfilled with n — 1 and k = 3. □ 



Proof of Theorem \1.5[ For p-groups, if the elements acting as bireflec- 
tions generate a proper subgroup, then this subgroup lies in a normal 
subgroup of index p. □ 
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Proposition 2.9. Suppose G has a normal subgroup N with factor 
group an elementary abelian p-group. If there exists a G G\ N whose 
fixed space in V is not contained in the fixed space of any bireflection in 
G\N, then no graded geometric separating algebra in k[V] G is Cohen- 
Macaulay. 

Proof. Without loss of generality, assume k = k. As G/N is an elemen- 
tary abelian p-group, there is a g G H 1 (G, k) with kernel N. The proof 
of [22l Theorem 3.9] provides a phsop ai,a 2 ,a 3 in kfV^]^ and a homo- 
geneous invariant h ^/Aniit^G^) such that hai G a/ 'Anni^jc (g) , 
for i = 1,2, 3. Hence, for some k > 0, the invariants a\ annihilate 
g' := h k g G H l (G, k[V]). Thus, the annihilator ideal Ann k ^ V ] G {9') 
has height at least 3. By Theorem 12.11 it now suffices to show that 
g' pm is nonzero for all m > 0. By [22, Proposition 3.5], we have 
Ann k [y]G(g pm ) = ^ Ann k [y]G (g) . Therefore, if g' pm is zero, then h kpm 
annihilates g p ™ , and so h G Ann k [ V }G (g) , a contradiction. □ 

Example 2.10. Let k be a finite field. For m > 3, set V = k 2m+1 , and 
consider the group G < GL(V) formed by the (2m + 1) x (2m + 1) 
matrices of the form 



/ 


Im+l 


o \ 














\ 




J 



where oto, . . . , a m G k, and I m denotes the mx m identity matrix. The 
group G is a p-group, and is generated by reflections, that is, by ele- 
ments whose fixed space has codimension at most 1 in V. Example 3.10 
in [22] shows that the hypotheses of Proposition 12.91 are satisfied, with 
iV the subgroup formed by the elements such that a m = 0, and a 
the element such that = 1 for all i. Hence, no graded geometric 
separating algebra in k(V] G is Cohen-Macaulay. < 

We end this section with a generalization of [221 Theorem 2.7]. 

Theorem 2.11. Let V reg be the regular representation of G over k. If p 
divides \G\, then every graded geometric separating algebra in k[V reg ] 
has Cohen-Macaulay defect at least \G\ Pj y- — 2. For \G\ > 5, this 
number is at least one. 

Proof. By [22| Lemma 2.6], there is a nonzero g G iJ 1 (C7, k[K- e p])- 
Lemma 12.121 implies all powers g pm are also nonzero. Since the fixed 
subspaces of elements of G of order p have codimension \G\{p — l)/p, 
the hypotheses of Lemma [2.71 are satisfied with k = \G\(p — l)/p. □ 

The regular representation of C4 was studied in Example 12.51 For 
G = C2, C7 3 , and C2 x C2, the invariant ring k[V^. e9 ] G is Cohen-Macaulay 
[22| Theorem 2.7]. Thus, these are the only groups such that there ex- 
ists a Cohen-Macaulay graded geometric separating algebra in k[VJ. e5 ] G . 
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Lemma 2.12. Let V be a permutation representation of G. If g in 
H 1 (G,k[V}) is nonzero, then g pm is nonzero for all m > 0. If, in 
addition, V is faithful, then every graded geometric separating algebra 
in k\V® k ] G has Cohen-Macaulay defect at least k — 2. 

Proof. As V is a permutation representation, there is a set of monomials 
M C k[V] such that k[V] = ® heM (Gh). Thus, if g e H\G,k[V]), 
there is a (finite) decomposition g = J2heM 9hi where each g h is in 
H l (G, (Gh)). For m > 0, we have the decomposition g pm = J2heM 9h > 
where is in H 1 (G, (Gh')), and /i' is the unique element of DM. 
If g is nonzero, then gy,, is nonzero for some h. As (Gh) and (Gh pm ) are 
isomorphic permutation representations of G, the element g v h is also 
nonzero. The additional statement follows by Lemma [2.71 □ 



In this section we concentrate on representations of the alternating 
group A4 over an algebraically closed field k of characteristic 2. The 
group Ai is the smallest modular group G for which the cohomology 
H 1 (G, k) is trivial. In order to apply Theorem l2.ll we must look outside 
the direct summand H\G,k) of H\G, k[V}). 

Pick x £ ^4? an element of order 2, and r, a 3-cycle, so that A 4 
is generated by \ and r. The unique Sylow 2-subgroup P in A A is 
generated by x an d r _1 xr. Let w be a fixed primitive third root of 

2 

unity in k. Define ^k and w k to be the one-dimensional representations 
on which P acts trivially, and r acts via multiplication by u and u 2 , 
respectively. Formally, x k denotes the trivial representation. 

Lemma 3.1. For i = 1,2, the element of if 1 (y4 4 ,^k) given by the 
cocycle x ' , T h_ is nonzero. 

Proof. Assume i = 1. Let V := (v±, v 2) be the representation given by 



This is an indecomposable representation [T5(, Theorem 7.0.3], with a 
submodule W := (vi) isomorphic to w k. Furthermore, A4 acts trivially 
on the quotient V/W, and so we have a nonsplit exact sequence 

_> w k V k 0. 

By [231 section 2], it follows that H\A^ W \) ^ 0. In particular, the 
cocycle given by x l— ^ r ^ is nontrivial. Similarly, for i = 2, the 
cocycle in Z 1 {A,i, u k) given by x ^ r ^ is nontrivial. □ 

Corollary 3.2. J/V k@W® k , where W is a faithful representation 
and i = 1,2, then every graded geometric separating algebra in k\V] Ai 
has Cohen-Macaulay defect at least k — 2. 



3. The Alternating Group A 4 
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Proof. Without loss of generality, i = 1. Since V* =^ 2 k©(W*) efc , there 
is a direct summand of k[V] isomorphic to S( u k). Since S 2 ™ ( w k) is 
isomorphic to w k or w k, Lemma 13.11 implies that the cohomology class 
g m G -H" 1 (^4, 5' 2m (' J k)), given by the cocyle x ^ uj2 " Z , T l— > 0, is 
nonzero for all m. As # m = g$ m , the result follows by Lemma 12.71 □ 

Exploiting the classification of the finite dimensional representations 
of v4 4 [7] (with the notation of pJ3, Chapter 7]), we obtain a much 
stronger result: 

Theorem 3.3. Suppose V is a faithful, indecomposable finite dimen- 
sional representation of v4 4 . 7/k[\^] A4 is non Cohen- Macaulay, then no 
graded geometric separating algebra in k\V] AA is Cohen- Macaulay. 

Remark 3.4. The indecomposable representations V of such that 
k[V} M is Cohen-Macaulay are listed in [T5J, Corollary 5.2.16]. In par- 
ticular, when dinikfV) > 7, k[V] 4 is non Cohen-Macaulay. 

Proof. Whenever k[V] A4 is non Cohen-Macaulay, the fixed space of 
any nonidentity element in P has codimension at least 3 in V (see the 
classification). Thus, by Lemma [2771 it suffices to find g G ^(A^, k[V]) 
such that g 2 ™ is nonzero for all m > 0. We use the classification to 
separate our argument into two cases. 

First, we suppose V is of the form W s (u) e ), where seZ(s may be 
negative only if it is odd), and < e < 2. The dimension of V is 
n := \s\. Let {t>i, v 2 , . . . , v n } be a basis of V such that At acts via the 
matrices given in (T5J Theorem 7.0.3]. In particular, the action of is 
upper triangular, and r acts diagonally via multiplication by third roots 
of unity. If k[K] A4 is non Cohen-Macaulay, the subset {fi,f2, • • • ,vi} 
is contained in V p , for some I > 2. Moreover for some r < I, we have 
rv r = u~iv r , where j = 1 or 2. Let {x±, ■ ■ ■ , x n } be the dual basis, 
so that k[V] = k[xi }. Define u := rio-eP^ 3 ^)- Then 

tu = Y\_( Ti7t1 ( TX r)) = TT ^'(cav) = ^u. 
aeP o-eP 

Since v r G V p , the set {xi,x 2 , ■ ■ ■ ,x r -i,x r+ i, . . . ,x n } spans a kP- 
submodule of V*, thus k[V] = k[xi, x 2 , ■ ■ ■ , x r -i, x r+ \, . . . , x n ] [x r ] as 
kP-modules. Therefore, by [151 Proof of Lemma 1.3.2], if M is a kP- 
module direct summand of S(V*), so is u ■ M, and by induction so 
is u l ■ M for any integer i. In particular, as 1 G S°(V*), we have a 
sequence (1), (u), (u 2 ), ... of kP-direct summands. Each (u l ) is a kAr 
direct summand, and = a; ' J k. The element ^ m G -£P(v4 4 , (u 2 " 1 )) 
given by the cocyle x l— *• c<j^ 2mj M 2m , r i— > 0, is nonzero by Lemma [3.11 
As g m = (/q" 1 , we are done. 

Second, suppose V takes the form W§d,\i where A G k U {00}, not 
a third root of unity, and d > 1, an integer, are such that if d = 1, 
then A is not 0, 1, or 00 (see [T5l Theorem 7.0.3] for notation). As 
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a kP-module, V* = V 2d , a © V 2d ,p © V 2dil % Section 4.3], where a, 
(3, and 7 are elements of k U {00} depending on A. The element r 
acts on V by permuting the three kP-module summands. As a kP- 
module, S(V*) S(V 2d>a ) ® 5(^/9) © 5(F 2rfi7 ), and therefore, 5(V*) 
has direct summands isomorphic to 5(V2d,a), S(V 2di p), and S^V^y). 
These summands are permuted by the action of r. Let {xi, . . . ,xe d } 
be a basis of V* such that the action of P is given in block form; 
that is, Xi,...,x 2d is a basis for the summand isomorphic to V 2di0t , 
and so on. Define u a := rio-eP^ 1 ' anc ^ u £ := YlaeP <TX2d + 1 ^ anc ^ 
m 7 := rio-eP ax 4d+i- By the same argument as before, for each integer 
i, (ttg) is a direct summand (with trivial P-action) of the kP-module 
S(V 2dta ), and hence of the kP-module S(V*). The analogue holds for 
up and u-y. Furthermore, we have 



Similarly, ru^ = and tu^ = u l a . Thus, U l := (u^UpjUt) is a kAr 
direct summand of S(V*) on which P acts trivially. Since [A4 : P] 
is odd, r acts diagonally with respect to some basis of IP. A short 
calculation shows that u := w 7 + ct'W/j + co ,2 m q , spans a direct summand 
of U 1 on which r acts via multiplication by lo, in other words, (w) ^k. 
boi each m, we nave that u = + u + a; u a spans a 

direct summand of U 2 " 1 isomorphic to ^ k. 

The element g m G H l (A 4 , (m 2 ™)) given by the cocyle \ ^ uj 2m+1 u 2m , 
r 1— > is nonzero for each m > 0. As g m = g 2 " 1 , the result now follows 
by Lemma 12. 71 □ 



Our results have shown that in many of the situations in which the 
ring of invariants is non Cohen-Macaulay, the same holds for any graded 
geometric separating algebra. Hence, one might wonder if the existence 
of a Cohen-Macaulay graded geometric separating algebra implies that 
the ring of invariants itself is Cohen-Macaulay. The following example 
shows this is not true. 

Example 4.1. Let G = C 2 x C 2 = (o~,r) be the Klein four group, 
and let k be a field of characteristic two. Consider the 5-dimensional 
representation of G given by 



By [TTI Theorem 7], the ring of invariants is not Cohen-Macaulay. 
In fact, [17] shows that there exists a nonzero cohomology class g G 
H x {Gi k[V]) whose restriction to each proper subgroup of G is zero. 



By pH Lemma 2.4, Proposition 2.5], ^/Arni k[v]G (g) = 3(V G ) n k[V] G , 




aeP CTGP 



4. Concluding Remarks 
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where 3(V G ) denotes the ideal of polynomials / G k[V] vanishing on 
V G . Thus, the height of Aim^ry-p^) is codim(V AG ) = 3 while, by [221 
Corollary 1.6], its depth is only two. 

Using MAGMA [4J and the methods of [2H Section 2] , one can verify 
that 

{ai := x 3 ,a 2 := x 4 , a 3 := x s , 

ct4 := x\ + x\x\ + x 2 x 3 x 4 + £1X3X4 + X\X 3 x\ + X1X3X4X5+ 

X x x\ + x\x\ + X 2 x\x h + X2X3X4, 

4 22 2 22 2 2 

a 6 := xfx? + X1X3X4X5 + X1X4 + x^xj, + X2x\x<z + X2X3X 2 ., 
a 7 := x 1 x\x 5 + 21X4X5 + x|x 3 x 5 + x^xf + 22X3X 2 + x 2 x|} 

forms a geometric separating set. Furthermore, {01,02,0-3,04,^5} is a 
hsop for the geometric separating algebra A := k[ai, a 2 , . . . , a 7 ]. As a 
module over k[ai, a 2 , a 3 , 04, a 5 ], A is freely generated by {1, a^, a 7 , a 6 a 7 }. 
Therefore, A is Cohen-Macaulay. As G is a p-group, by Theorem 11.51 
G must be a bireflection group, which is indeed the case. The Hilbert 
Series of AisH(A,t) = (1 ^ 2 3 f ff )2 • Since 1 ft) = (-l)H 3 H(A,t), 
we even have that A is Gorenstein, but not strongly Gorenstein, as 
3 ^ dim V. 

Note that since k[V] G is integral over A, the height of Ann^(^) is 
also 3, and as A is Cohen-Macaualay, the depth of Ann^g) must be 3. 
Theorem 12.11 implies that there exists a p-power q such that g q = 0. < 

We end with an example which shows that even in the non-modular 
case, the good behaviour of separating algebras is not guaranteed by 
that of the invariant ring. 

Example 4.2. Let G = C4 be the cyclic group of order 4, and let k be a 
field of odd characteristic containing a primitive fourth root of unity £. 
Consider the 2-dimensional representation V of C4 = (a) given by 
a ^ (gg). If k[V] = k[x,y], then kfl/f 4 = k[x 4 , x 3 y, x 2 y 2 , xy 3 , y 4 }. 
On points where x 4 is zero, the function x 2 y 2 also takes the value 
zero; on any other point v, we have x 2 y 2 {v) = ^ffep ■ Thus, the 
value of x 2 ?/ 2 as a function is entirely determined by the value of x 4 
and x 3 y. Therefore x 4 , x 3 y, xy 3 , y 4 form a geometric separating set and 
A : = k[x 4 , x 3 y, xy 3 , y 4 } is a geometric separating algebra. Note that A 
is not Cohen-Macaulay, since the hsop x 4 ,y 4 does not form a regular 
sequence (despite being coprime) 0, Exercise 2.1.18]. 

On points where x 4 is zero, xy 3 is zero, and on any other point v, 
xy 3 {v) = ^ x 4^))2 • Thus, the hypersurface k[x 4 , x 3 y, y 4 ] is also a graded 
geometric separating algebra. Therefore, we have a Cohen-Macaulay 
graded geometric separating algebra, inside a non Cohen-Macaulay 
graded geometric separating algebra, inside a Cohen-Macaulay invari- 
ant ring. < 
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